Extended Markup Language (XML) is a markup language by which a definition may be assigned to one or more words in a document by using tags. In order to assign a definition to a phrase in a document, a tag pair, which includes a start tag, placed at the beginning of the phrase, and a closing tag, placed at the end of the phrase. FIG. 1 shows a front view of a document (9) in which text items are delineated by XML tag pairs. The document (9) may be, for example, part of documentation that explains the operation of a tool. An outer tag pair is represented by a start tag (10) and a closing tag (12), which define a CUSTOMER. A first nested tag pair defines an ID NUMBER for the CUSTOMER, with a start tag (14), a closing tag (16), with an ID NUMBER of 234 (18). A second nested tag pair defines a NAME, with a start tag (20), a closing tag (22), and a NAME of “John Doe.”
Documentation associated with a tool (e.g., a command line tool, such as a Bash shell), a graphical user interface tool (a GUI tool), an operating system, a debugging tool, web applications, applets, custom software applications, etc.) can take multiple forms. Common forms of documentation include electronic reference texts, on-line help files, man pages, reference manuals, comments within source code, etc. Documentation may come in a variety of forms, such as computer files, which may be accessible through a web browser, editor, etc., or documentation may be in hard copy form, such as a book or technical manual.
Documentation may help explain a particular source code statement, explain a particular tool command or functionality, or instruct a user as to the function of the tool. For example, documentation for a GUI tool may explain to a user how to manipulate or use various widgets of the GUI tool, such as buttons, text boxes, dialog windows, etc. For example, a sentence in GUI tool documentation may display instructions telling the user to, “copy the contents of text box A into text box B, and hit ‘Enter’.” The GUI tool documentation may then display a result of the user performing the instructions, such as showing a picture of a window with the copied text. Likewise, command line tool documentation may display instructions telling the user to type a word or words. For example, the command line tool documentation may instruct the user to type “grep it file1” into the command line tool. The command line tool documentation may then show a result of the instruction, such as displaying a line or lines of text containing “it.”
Several standard testing tools are used in testing of command line tools and GUI tools. One testing tool is a test suite, a collection of files of one or more tests used to determine whether a tool, e.g., GUI tool or a command line tool is functioning as specified. A test used within the test suite typically contains one or more inputs, as commands, to the tool being tested, designed to exercise one or more of the tool's features. A related testing tool is a test harness, which uses a test suite to test the operation of a tool being testing by accessing the test suite and applying the tests to the tool being tested. The test harness evaluates the response of the tool being tested to determine whether the tool being tested is working properly. Another component used in testing is a golden file. The golden file typically contains a pre-determined proper response of the tool being tested.
Examples embedded within tool (either command line tool or GUI tool) documentation may contain errors. The errors in such embedded examples may exist for a variety of reasons. One possible reason for the errors is improperly written code statements shown in command line tool documentation. Also, an image shown in GUI tool documentation as a result of a user action may not be the same as the image that actually is displayed when the user performs the user action. Changes in a system or a network configuration, etc., may also cause errors. Another source of errors is when a developer, software engineer, etc., changes the tool (either intentionally or unintentionally) in a way that is inconsistent with the tool's specification. Thus, the user may take some user action, and expects the particular response as documented in the tool documentation, but gets a different response from the tool. Such errors often result in frustrating the user of tool documentation.
FIG. 2 shows a sequence of operations to test embedded examples in command line tool documentation. Testing embedded examples in command line tool documentation is shown in FIG. 2 and described below.
A tag set is created (Step 50). The tag set includes tag pairs, such as, <test-example>, </test-example>, <test-step>, </test-step>, <test-name>, <test-name>, <test-source>, </test-source>, etc. Embedded examples in the command line tool documentation are tagged by delineating pertinent words or phrases in the command line tool documentation using tag pairs from the tag set (Step 51). Tagging the embedded examples creates extractable embedded examples. For example, for a command line tool whose main function is to accept a word typed via a keyboard, reverse the word, and print a reversed word to a display device, the command line tool documentation includes an embedded example that instructs the user, as follows:                Example 3.2.1. Type “reverse backwards” at the command prompt and hit ‘Enter.’ You will see the word “sdrawkcab” displayed on the screen.        
The embedded example above is tagged using tags pairs from the tag set to delineate pertinent words and phrases, as follows:                <test-example> . . .        Example <test-name> 3.2.1.</test-name> Type “<test-step> reverse backwards </test-step>” at the command prompt and hit ‘Enter.’ You will see the word “sdrawkcab” displayed on the screen.        <test-golden-file        location=‘http://www.docs.com/examples/program.output’></test-golden-file>        . . . </test-example>        
The <test-step> tag pair shown above delineates the pertinent phrase “reverse backwards,” which represents a command given by the user to the command line tool. After tagging, example 3.2.1 above is an extractable embedded example. The <test-golden-file> tag pair above gives a location for a correct result of the test, i.e., what the user expects to see when executing the instruction (Alternately, instead of using the <test-golden-file> tag pair, “sdrawkcab” could be tagged a using a <test-output> tag pair). Additional tags may be used as needed. For example, <test-source> and </test-source> tags may be used to locate additional files, as needed.
Once embedded examples in the command line tool documentation are tagged, and extractable embedded examples are created, as described above, the extractable embedded examples are extracted from the command line tool documentation (Step 52). A test suite is created using extracted examples (Step 54). The test suite includes a number of inputs that test functionality of the command line tool. For example, referring to the previous example, an input to the command line tool, and thus a part of the test suite, is a string “reverse backwards,” extracted as an extractable embedded example from the command line tool documentation, where the command line tool is expecting a command (“reverse”) and an argument (“backwards”). With the test suite created, a command line tool is selected against which to execute the test suite (Step 56). The test suite is executed against the command line tool (Step 58). For example, operating system calls may be used (along with other mechanisms/techniques) to execute the test suite.
A result of executing the test suite against the command line tool is compared to a stored result in a golden file (Step 60). For example, a correct command line tool output in response to the test from the previous example is “sdrawkcab.” The result of the test is output to, for example, a computer file for storage (Step 62). If output resulting from the test does not match the golden file, a determination is made as to whether a correction to the command line tool documentation is necessary and/or appropriate (Step 64). For example, a specifications guide, which includes criteria for proper operation and expected performance of the command line tool, and is linked to the command line tool documentation, may be used in Step 64. If a correction to the command line tool documentation is necessary and/or appropriate, then extractable embedded examples within the command line tool documentation are corrected (Step 66). For example, a test harness, using the specifications guide, may use text substitution to correct extractable embedded examples in the command line tool documentation. Otherwise, if corrections to the command line tool documentation are not determined to be appropriate, corrections are not made programmatically, but are made by a documentation writer, a test engineer, etc.